NEW YORK (CNNMoney) -- There are far more jobless people in the United States than you might think.

While it's true that the unemployment rate is falling, that doesn't include the millions of nonworking adults who aren't even looking for a job anymore. And hiring isn't strong enough to keep up with population growth.

As a result, the labor force is now at its smallest size since the 1980s when compared to the broader working age population.

"We've been getting some job growth and it's been significant, but it hasn't yet been strong enough that you start to get people re-engaging in the labor market," said Keith Hall, a senior research fellow at the Mercatus Center and former commissioner of the Bureau of Labor Statistics.

A person is counted as part of the labor force if they have a job or have looked for one in the last four weeks. As of April, only 63.6% of Americans over the age of 16 fell into that category, according to the Labor Department. That's the lowest labor force participation rate since 1981.

It's a worrisome sign for the economy and partly explains why the unemployment rate has been falling recently. Only people looking for work are considered officially unemployed.

Jason Everett, for example, wouldn't be counted.

Out of work for nearly three years now, Everett has given up his job search altogether.

Instead, the unemployed plumber and Air Force veteran takes a few community college courses and looks after his two children while his wife is the primary breadwinner.

"I'm not even totally convinced the college degree is really going to help at this point, but I figure at least I'll be doing something," he said.

Last year there were 86 million people who didn't have a job and weren't consistently looking for one, according to Labor Department data.

Older people, ages 65 and over, account for more than a third. Young people between 16 and 24 make up another fifth. More than half don't have a college degree and more than two thirds are white.

Many of the teens and 20-somethings may be enrolled in either high school or college full-time. And many of the over 65 crowd are probably retired.

But what about the other 36 million folks who fall in between?

The truth is, the Labor Department simply doesn't know why they're not in the labor force. Many may be staying home with children or other relatives. Some may have gone back to school or retraining programs. Others could be disabled and unable to work, and some may have retired early.

"Even in the best of times, there are millions of people who don't want to work for a variety for reasons," Hall said.

But he suspects the number of "disengaged" Americans, like Everett, is higher than usual as a direct result of the recession.

About six million people claim they want a job, even though they haven't looked for one in the last four weeks. If they were to all start applying for work again, the unemployment rate would suddenly shoot up above 11%.

"At this point, the labor market is worse than people realize because people are discouraged. Certainly, a large number of workers have given up on the job market," Hall said.

That said, the decline in labor force participation is not a new problem. After peaking at 67.3% in early 2000, the rate has been falling ever since.

Researchers at the Chicago Federal Reserve attribute a large part of the decline to the recent recession and lackluster recovery, but the other half to long-term demographic trends.

For example, as more women entered the labor force between the 1960s and 1990s, the participation rate rose rapidly. That effect may have plateaued since then.

Meanwhile, as Baby Boomers entered their prime working years, they also drove the participation rate higher. Once they started hitting their 50s and 60s though, many started transitioning into retirement.

Finally, teenage jobs have been on the decline and college enrollment picked up in the last decade, leading more young people to not be counted in the labor force.

As these trends continue, the Chicago Fed expects the labor force participation rate will keep falling, hitting 62.4% by 2020.

That poses a problem for a variety of reasons.

It hits tax revenue and makes it harder to fund social safety nets like Social Security. Not to mention, it's likely to increase income inequality.

Most importantly though, it makes the U.S. economy less productive and weighs on growth.