Luke Grimes, who plays son Kayce Dutton to Costner’s John Dutton III in the Paramount drama series, recently spoke to Esquire about the temperature on set after Costner’s character was killed off following five seasons on the hit show.

“Hopefully everyone can see that it was time,” Grimes, 40, told the outlet in an interview published Tuesday, Dec. 10.

“To be really honest, there was a part of Kevin being gone that meant some of the conflict was gone. Obviously, it didn’t make it super fun to be around. Not pointing any fingers, but it was actually the easiest season we’ve filmed,” he said of the show’s season 5’s second half. Costner departed the show after season 5’s first half.

Much of the conflict Grimes was referring to had to do with Costner, 69, and series creator Taylor Sheridan not seeing eye to eye amid the actor working on his own Western epic, Horizon: An American Saga, which saw an overlap in filming time.

“I don’t want to get down in the gutter with the Yellowstone thing but what I’m telling you is straight up. I have taken a beating from those f—— guys and I know a lot of times where it’s coming from. I just elected not to get into that,” Costner previously told Deadline

“I made Yellowstone the first priority, and to insinuate anything else would be wrong,” he said in the May interview.

Despite the drama, Grimes added that Yellowstone had naturally run its course.

“It feels like good timing,” he said of the series coming to an end. “Seven years is a long time to do anything, and I had never played a character for more than a few months before.”

The 50 Shades of Grey alum added that there were still good times: “I’ve loved all of it.”

 “But there is a part of it that — once you read those final episodes — sees that there’s a finality to it. There’s always talks of spin-offs,” Grimes said, noting that he’d be open to doing one.

“I’d do it [but] I just don’t understand how it would work once the story ends. Kayce wants to cowboy and be happy with his family. He doesn’t want to kill people anymore. He doesn’t want the weight of a huge mega ranch that isn’t sustainable in today’s times. He wants his little slice of heaven. It’s that simple,” he said of his character’s fate.

Grimes’ comments to Esquire about Costner’s exit echoed statements he previously told PEOPLE.

“Him not coming back felt like, ‘Okay, we’re going to have to land this plane now for real,’ ” Grimes told people of the storyline.

“I think the patriarch leaving was always going to be part of the story. That’s always where it was headed is like, what do these kids do? What does this family do when their rock is gone?”

He also admitted that it “felt a little different” not having Costner around, calling the cast “a big family now.”