When your olive trees start producing fruit, it's time to harvest the olives. Wait until the olives are fully ripe and have turned a dark purple or black color. Gently handpick the olives, being careful not to damage the branches or leaves. Once harvested, you can use the olives to make olive oil, or preserve them for culinary use.