Much of the debris in the near-surface ocean collects in so-called garbage patches where, due to convergence of the surface flow, the debris is trapped for decades to millennia. Until now, studies modelling the pathways of surface marine debris have not included release from coasts or factored in the possibilities that release concentrations vary with region or that pathways may include seasonal cycles. Here, we use observational data from the Global Drifter Program in a particle-trajectory tracer approach that includes the seasonal cycle to study the fate of marine debris in the open ocean from coastal regions around the world on interannual to centennial timescales. We find that six major garbage patches emerge, one in each of the five subtropical basins and one previously unreported patch in the Barents Sea. The evolution of each of the six patches is markedly different. With the exception of the North Pacific, all patches are much more dispersive than expected from linear ocean circulation theory, suggesting that on centennial timescales the different basins are much better connected than previously thought and that inter-ocean exchanges play a large role in the spreading of marine debris. This study suggests that, over multi-millennial timescales, a significant amount of the debris released outside of the North Atlantic will eventually end up in the North Pacific patch, the main attractor of global marine debris.