Farmers stop working

[The Pilgrim]

So here is my problem. I build many farms.  Many many farms. Most of them consist of 2 7x15 fields, a barn and a house, all surrounded by a fence. Works great for a while, but the all of a sudden when I get to around 100 people or so, the farmers just stop working. They'll plant super late in the season then take thier sweet time harvesting leaving most of the crop to die in the field. Why?