From Lakoff and Johnson's Philosophy in the Flesh:
What could be simpler or more obvious than colors? ... We see colors as inhering in things. Blue is in the sky, green in the grass, red in the blood, yellow in the sun. We see color, and yet it is false.... Another thing to bear in mind is that light is not colored. Visible light is electromagnetic radiation, like radio waves, vibrating within a certain frequency range. It is not the kind of thing that could be colored. ...  Our color concepts, their internal structures and the relationships between them are inextricably tied to our embodiment. They are a consequence of four interacting factors: lighting conditions, wavelengths of electromagnetic radiation, color cones, and neural processing... Colors are not objective; there is in the grass or the sky no greenness or blueness independent of retinas, color cones, neural circutry, and brains. Nor are colors purely subjective; they are neither a figment of our imagination nor spontaneous creations of our brains.
What could be more obvious than the fact that the past is different from the future? Time seems to have a direction, a way in which it passes or flows. This isn't just a feeling- we can observe ourselves and other systems in the universe and note that their past states are different from their future states in asymmetric ways. However, quantum field theory, the most accurate physical theory we've ever come up with, makes no distinction between past and future: if you take time and reverse it, you get the same equations.* The conflict between these two observations leads to the problem of the "arrow of time."

The most common answer to this problem is taken by pointing at the fact that we observe entropy increasing over time. Entropy then provides the arrow. But in what sense do we mean this? It doesn't make sense on the largest possible scales. If we consider the universe to be one giant wavefunction evolving according to a unitary operator (as most cosmologists do), then entropy of the universe must remain constant. However, we clearly observe entropy increasing over time- macroscopically irreversible processes are all around us. We can only get these two things to mesh if take into account the particular ways in which humans constitute a subsystem of the overall universe.

In fact, the very notion that the arrow of time seems to be related to increasing entropy should set off alarm bells that it has to do with humans as a subsystem of the universe, rather than
Despite the fact that entropy is an unavoidable and hugely important tool in physics, it is important to remember that it is a measure on probability distributions. And as every good Bayesian knows, probabilities are subjective. Entropy only makes sense in terms of an uncertain distribution over possible world-states. This makes entropy (and by extension, temperature**) different from other thermodynamic variables such as energy, pressure, and volume, which can be defined on individual microstates- that is, actual states of the world. Because thermodynamics is so powerful and has lasted so long (it survived both the relativistic and quantum revolutions, after all), entropy seems like part of the territory, but it is really part of the map. It is just such a general part of every possible map that it seems to be a first-class ontological citizen.

Subjective time isn't completely arbitrary or independent from objective time- after all, people tend to have fairly consistent judgments of duration. But neither is it completely determined by objective time. In this sense, I think we can make a useful comparison between color and time. Some of the structure of color correlates with the structure of light, but some doesn't. In our minds, colors can be arranged in a wheel, like this:

Note that on the color wheel, red is next to purple and the two blend smoothly into each other, despite the fact that the wavelengths of light corresponding to those colors are on opposite ends of the visible spectrum. This is a structure of color that does not exist in light in the real world. Similarly, we can think of the arrow of time as a temporal structure that does not exist in the real world.

The English language conveniently has separate words for "color" and "light" that can allow us to easily disambiguate between the experience we have of the world and the thing in the world that is most directly linked to the sensation. Unfortunately in the case of time, we use the same word for both.

--------------------------------------------
*More accurately, quantum field theory implies CPT symmetry- that is, if you reversed the arrow of time, reflected everything in a giant mirror, and changed particles to antiparticles, everything would look exactly the same. But this is close enough for our discussion.

** In statistical mechanics temperature is defined by $\frac{1}{T} = \frac{dS}{dE}$, where $T$ is temperature, $E$ is energy and $S$ is entropy. Temperature is not, as is almost universally taught, a measure of the average energy of particles in a system. That is measured by the energy (unsurprisingly). Temperature is better seen as a parameter dictating what sort of probability distribution you should place over a collection of possible states given that a system is in thermal equilibrium. The reason for the confusion is that in many common systems energy is directly proportional to temperature. I may write a blog post clarifying this in the future.