In a series of posts several years ago I argued that Millard Fillmore's run for the presidency in 1856 under the American Party (Know Nothing) banner had nothing to do with nativism or anti-Catholicism. Millard was using the Know Nothings as a vehicle to try to forge a pro-Union party - think of it as a precursor to John Bell's run in 1860 as the candidate of the makeshift Constitutional Union Party.
In Parties, Slavery, and the Union in Antebellum Georgia, Anthony Gene Carey makes clear that Georgian supporters of Fillmore, at least, backed his candidacy because they understood Unionism to lie at its core:
About the illustration, entitled The Great Presidential Race of 1856:
Georgian Americans . . . offered [Millard] their nomination at a July  state convention. Fillmore, in graciously accepting the endorsement, quizzically noted that he was already the national American candidate, but his Georgia backers adamantly refused to regard him as such. Fillmore was only "nominally the candidate of a party"; he was really "the candidate of the people." The Fillmore movement was "an insurrection of the honest masses against the despotism of party and party leaders." The "proud and independent" Georgian Americans rejoiced that they and their candidate were untrammeled by platforms "framed in Northern latitudes." Determined to mold Fillmore in their own image, [Georgian] Americans created a proslavery, prosouthern Unionist who deplored partisanship but nonetheless loathed the national Democracy.
An animated comic scene ridiculing the Democratic and American party candidates. In the foreground is a somewhat rickety wooden "Democratic Platform," into which presidential candidate James Buchanan has just run, knocking his mount (running mate John C. Breckinridge, in the form of a buck) unconscious. Buchanan (center, dressed as a jockey) holds his right shin and curses a ragged black youth who stands laughing on the platform, "You infernal Black Scoundrel, if it had not been for you and that cursed Slavery Plank that Scared and upset my Buck, I should have won this race certain." The black youth jeers, "Ya! Ya! Ya! Why Massa Buck. Dis is de Democratic Platform, I tink I misunderstand you to Say dat you like dis Plank [. . .] in fac dat you was de Platform [. . . .]" One of the planks in the platform is labeled "Slavery," and another "Cuba," referring to the apparent proslavery and annexationist interests of the Democrats. On the far right a small boy is hoisted onto the back of a ragged, weed-chewing man with a beard. The urchin holds a flag reading, "We Po'ked em in 44, We Peirce'd em in 52 and We'll "Buck em" in 56." The bearded man looks at the child irritably and scowls, "Hello there!! are you a Fre'mounter." At left, American party candidate Millard Fillmore, riding a goose with the head of running mate Andrew Jackson Donelson and holding a "Know Nothing" lantern, cries, "I'm "All right on the Goose," and yet I dont seem to make much head way, Gentlemen you may all laugh, but if I'm not the next President the Union Will Be Disolved, The South Wont Stand It." In the background a crowd watches the race. Two men converse, saying, "Little too much Squatter Sovereignty about that Goose Fillmore?" and "A decided Curvature of the Spine, no Back Bone Sir, "All Dough" Sir, ha! ha! ha!" The majority of the spectators, however, cheer on Republican candidate John C. Fremont whose horse takes the lead at right. On an observation or judging deck nearby stands Brother Jonathan, holding what appears to be a timer's watch. The apt comic mise-en-scene and development of the minor characters here are characteristic of John L. Magee's work.