Here comes the hard part.

After waiting nervously while Aaron Judge sat out three straight games before being diagnosed with a stress fracture in his rib, the Yankees faced the last-place Red Sox in The Bronx on Friday night — and went out and lost for the third time in four games without their captain, this one 5-3.

This loss wasn’t all on the offense, as Ryan Weathers was tagged for five earned runs for the third time in his last four starts.

But the Yankees’ chances for a comeback against Boston were certainly diminished without Judge.

“It’s a big presence not to have in our lineup,” Ben Rice said. “It’s definitely gonna hurt us, but all we can do is keep moving forward.”

Facing former Yankee bust Sonny Gray — who brought a career 6.06 ERA at Yankee Stadium into the game — the offense never got going, as hitters No. 3-5 in the lineup went hitless in 11 at-bats.

“I thought for the most part we had quality at-bats,” manager Aaron Boone said. “We had a couple of opportunities to get a big hit.”

Rice gave the Yankees a lead in the first inning with his 18th homer of the season — a solo shot to right.

Asked if he thought he’d be pitched differently without Judge’s protection behind him, Rice said, “It’s not for me to say. We’ll see what happens. All I can control is the pitches I swing at and don’t swing at.”

But Weathers faltered in the second, loading the bases with one out.

Boston tied the game on a Wilyer Abreu grounder to short.

Willson Contreras followed with a chopper in front of the plate that went for an infield single to score Jarren Duran and give the Red Sox a 2-1 lead.

• CHECK OUT THE LATEST MLB STANDINGS AND YANKEES STATS

They added to it in the fourth, as Weathers gave up a one-out homer to Andruw Monasterio to make it 3-1.

Weathers said he’s been unhappy with his four-seam fastball and that one ended up in the seats.

The Yankees got a run back in the bottom of the inning on Spencer Jones’ double down the right field line that drove in Jazz Chisholm Jr.

But Weathers’ long-ball problems continued in the fifth, as Contreras hit a two-run shot into the second deck in left for a 5-2 lead.

He’s given up seven homers in his last four outings.

A Trent Grisham blast to right with two outs in the fifth got the Yankees back to within two runs.

They were on the verge of getting back in the game in the seventh, with Ryan McMahon on first and Rice facing lefty Danny Coulombe.

Rice got the count full, with lefty-killing Paul Goldschmidt on deck, but Rice whiffed to end the inning.

In the ninth, against another disappointing ex-Yankee, Aroldis Chapman, the Yankees got a leadoff walk from Max Schuemann, pinch hitting for Jones.

Anthony Volpe struck out looking on three pitches and Amed Rosario, pinch hitting for McMahon, also walked on four pitches.

Then came their third pinch hitter of the inning, José Caballero for Austin Wells.

With runners on first and second and one out, Caballero popped out to right and Grisham grounded out and the Yankees dropped their first game of the year to Boston after sweeping three at Fenway Park in April.