The word is best pronounced off-en, not off-ten. The t is silent. You would not dream of pronouncing soften as soff-ten. Nor would you consider it correct to voice the t in listen, chasten, or fasten. It is a mystery why so many prominent individuals nevertheless habitually say off-ten.
Yes, dictionaries typically include off-ten as an alternate pronunciation. But their mission is predominantly descriptive rather than prescriptive; they report actual usage (if sufficiently widespread) without necessarily endorsing its correctness or desirability. And yes, there may even be historical justification for voicing the silent t in often (though no more so than in soften or fasten or listen). But if you’re inclined to seize on your favorite dictionary, or advert to historical antecedents, to validate your preference for off-ten, just keep in mind what Bryan Garner says (in Modern American Usage): off-en, in his view, is “the educated pronunciation”; only “the less adept say” off-ten.
Unless you want insufferable language elitists like Garner (and me) to consider you “less adept,” stick to the silent t.