[tex]\displaystyle\int\tan^5x\,\mathrm dx=\int\tan^4x\tan x[/tex]
[tex]=\displaystyle\int(\sec^2x-1)^2\tan x\,\mathrm dx[/tex]
[tex]=\displaystyle\int(\sec^4x-2\sec^2x+1)\tan x\,\mathrm dx[/tex]
[tex]=\displaystyle\int\sec^3x\sec x\tan x\,\mathrm dx-2\int\sec^2x\tan x\,\mathrm dx+\int\tan x\,\mathrm dx[/tex]
For the first integral, let [tex]a=\sec x[/tex] so that [tex]\mathrm da=\sec x\tan x\,\mathrm dx[/tex]. For the second, [tex]b=\tan x[/tex] so that [tex]\mathrm db=\sec^2x\,\mathrm dx[/tex]. The last integral is standard, but if you're not familiar with it, you could write [tex]\tan x=\dfrac{\sin x}{\cos x}[/tex] and set [tex]c=\cos x[/tex] so that [tex]\mathrm dc=-\sin x\,\mathrm dx[/tex]. Now
[tex]=\displaystyle\int a^3\,\mathrm da-2\int b\,\mathrm db-\int\frac{\mathrm dc}c[/tex]
[tex]=\displaystyle\frac{a^4}4-b^2-\ln|c|+C[/tex]
[tex]=\displaystyle\frac{\sec^4x}4-\tan^2x-\ln|\cos x|+C[/tex]