Place cut fingerling potatoes in a large pot with a generous pinch of kosher salt and fill with enough water to cover by 1-inch. Bring to a boil and cook until potatoes are tender when pierced with a knife, about 8-10 minutes, depending on size. Drain into a colander and let cool slightly.
While potatoes are boiling, make the vinaigrette. In a large bowl, whisk together olive oil, vinegar, and mustard until emulsified. Stir in herbs, shallot, garlic, 1 teaspoon kosher salt and capers until combined.
Add sliced red onion and warm, drained potatoes to the vinaigrette and gently toss to coat. Let stand for potatoes to soak up vinaigrette and allow flavors to develop, at least one hour. Just before serving, fold in baby arugula leaves. Season to taste with salt and pepper. Salad is best enjoyed at room temperature.