1. Soluble chemicals
OK, we'll start at the top. Everything you're dealing with in a tank of water is to do with ions. Ions are either atoms or molecules of a chemical substance which have an imbalance of charge so that they're either positive or negative - how an individual 'species' (general term for an atom or molecule) behaves depends on what it's made of. Most chemical compounds split into ions in water and this is how they become dissolved - Intact species can't exist in solution but ions can quite happily interact with water molecules due to their charge.

1.1 Water ions
As well as the chemicals you put in, water also does the same thing. At any one time, a small proportion of a body of water will be split into H+ ions (positively charged hydrogen) and OH- ions (negatively charged oxygen with a hydrogen atom attached to it). These are what allow other chemicals to dissolve. Which leads us on to...

2. Acidity
First, let's dispel a myth. There isn't really any such thing as alkalinity. The term alkalinity is an academic crutch for describing a substance that is less acidic than pure water. Everything should really be considered in terms of acidity.
Acidity is quite simply the proportion of H+ ions in a solution. Its common notation, pH, means just that - proportion of H. Water has a pH of 7, which corresponds to 10-7 ppm, or parts per million of H+ ions. This assumes that the water is at a fixed temperature and pressure and is completely pure - pH can vary enormously with even tiny changes. Even distilled water on standing in air will become slightly acidic as carbon dioxide dissolves into it.
Some, or indeed most compounds when they dissolve in water will change its pH. What happens is that the positive ions of the compound will interact with the OH- and H+ ions from the water and so change the dynamic between H2O species and its ions. The upshot is that the body of water will produce more or less of its own ions to restore the balance.

2.1 Alkaline minerals (Bases)
Yes, I know, I know - I complain about something then adopt it myself in the same article. Still, it's a convenient reference to materials which increase pH. This sounds like an odd contradiction as when pH increases, the acidity decreases. In simple terms, alkaline minerals, or Basic minerals, remove H+ ions from solution. The common ones, carbonates, react with the hydrogen ions to produce carbon dioxide, which is released, and water, which returns to the system. As this happens continually it lowers the proportion of H+ ions in the solution and hence, the pH rises - the solution is less acidic than water.

12.2 Acidic minerals (Acids)
And correspondingly, we have acids. Acids release H+ ions into a solution. A good example of this is the nitrogen cycle, explained in detail in the wonderful article by ParrotFishRule. Basically, the chemical processes that convert the nitrogenous wastes release H+ ions due to the fact that they use oxygen ions from the water - leaving the rest of the molecule - H+ ions - behind.


3. Hardness
Hardness is a rather complicated area dealing with the quantities of different types of ions dissolved in the water. Of the two types, there is General Hardness (GH), which refers to the total quantities of ions dissolved in the water and Carbonate Hardness (KH - the standard adopted is German, in which 'carbonate' is spelt with a K...) which deals with the proportion of Hydrogencarbonate ions, HCO3-. General Hardness is also referred to as Permanent hardness because it's very difficult to change. For the purposes of aquaria, it's best left alone as the only safe and reliable method of changing it is 'cutting' with pure water. Carbonate hardness however is extremely important...

3.1 Buffering
Carbonate hardness is important because it dictates your tank's Buffering Capacity. Buffering capacity is the ability of a given body of water to resist changes in pH. Because the hydrogencarbonate ions in the water are a base, they can react with H+ ions and remove them from the system. With a given value of carbonate hardness, your water will have a fixed pH, but how much that pH will change when new chemicals are introduced depends on your buffering capacity. If the amount of new H+ ions released by the chemical is small compared to the amount of carbonate hardness, the pH will remain relatively constant, which is what you want. If the carbonate hardness is low, a small amount of new acid will make a large change to the body's pH.
In general, carbonate hardness is produced by your water coming from an area rich in chalk and/or limestone, which releases Carbonate ions into your water. Such water will normally have a high GH and KH, and will also tend to have a higher pH. Areas which take their water from reservoirs rather than groundwater, or are on clay or peaty soils, will tend to have a lower GH and KH, though in this case GH may not change much while KH is reduced by reacting with the acids in the soils. This water will also tend to have a low pH and a low buffering capacity.