Custom Systems - How do you do them well?

What I'm asking is:
I want to make a game using a custom system and I'm still trying to work out the mechanics. I'm currently wondering if there's a market for custom systems and, if so, do people like complex or simple, class-based or free-to-pick or etc.? Do people prefer a specific amount of choice or..? And just overall, are there any tips you system-making veterans can give me?
Thanks!
EDIT: I've DM-ed some TTRPGs before, and also played a wide variety of them, from DnD to Shadowrun to Fate, so the actual "doing it" isn't really the problem, but the creative aspect.

Depends on what you mean with a market for it. If it's well made and interesting you should be able to find some people who are interested. If you want to get rich from it, you should probably look at another product.

There are different trends. Many people currently like simpler systems but there's also a market for more complex games...
To pretty much all your questions, I would just say that there are people who like it either way. You just have to decide what you want for your game, stick with it and do it well.

I use a mix. Though on PBP I prefer simple game mechanics. I am using Barebones fantasy, tiny dungeons and fantasy onedice for the games I run.

Still I like games with crunch. One of my best campaigns was using rolemaster. we played twice a month for 4-6 hours for 3 years using it.I find it is best to pick the system that best suites the game you are going to run.

I have played in my life over 50 different systems in my 40 years of rpging and all have pros and cons, so do what you think is best.

As to making systems I have done that a few times, I usually find a system that is close to it and then modify it to fit what I want. So at first usually is just houeruling a system but over time it turns into the system that i want to use. There are a great many generic systems to start look at to see where to base your own with. Stealing from various ones helps as it means you don't have to start with nothing to begin with.

Any of those generic systems you'd recommend?

A couple to look at to see a variety in order of complexity in my eyes are ondice, d6, fuzion, wion, and Gurps.

If you are starting I would think D6 is a good place to start. Fuzion has a bad wrap but it has a good many thing to take from it (I prefer the interlock system that it was based off of). the best of all both those systems you can get for free.

Another system that is good is the Ages system (there is a fantasy and also a modern version of it as well as a dragon age game based on it.)

I'll have a look at D6 to start, then, and go down the list. Thanks for the help!

From what I've seen (online) you may want to consider defining the setting first and only then picking the rules that fit that setting, instead of going for general catch all systems. As far as I understood, that makes the game a more unique experience, and there seems to be a "market" for that.

Yeah, I'm not gonna straight up use whatever system I pick, I'll take the parts I don't like and adjust them, along with customized stats and stuff to fit the setting

Market, well it depends what you mean. there are plenty of people selling their custom systems on drivethruRPG, but not all of them are popular. I'd say having a good system is one thing, but a good presentation also helps a lot.

As for checking rules lite games, I have a resource thread here where you can discover many of those games for free or really cheap.

After going through many of those RPGs, I'm liking some of the rolling systems from d6, some of the stats from Fuzion and the action system of Twilight (albeit more turn-based) and so I might try and use those to make something unique

I think the best advice I've seen regarding making your own RPG is:

1. Don't do it for the money/expecting to ever make a profit.

2. Don't do it to try and make something popular.

3. Do it because you have a vision for a game that does something that no other system does. Everything else will flow from that, if it's gonna flow.

Trying to piece together mechanics you like from systems you have no practical experience with is probably not a good way to go about. It's very likely those disparate systems had disparate goals when they were created and so those disparate pieces might not at all fit together.

Instead, first work to understand your subject. Once you have that understanding (where, "not having to ask for advice from internet strangers on where to start" might be a good metric of when you have attained understanding), work from that point of understanding toward your clearly stated and consistent goal or vision for your game.

Not trying to be rude, but presently it sounds like you're fumbling around in the dark...

EDIT: spelling is hard (in the world of autocorrect)

Trust me, I'm hardly doing it for profit or to become popular. I just want to make a system to go along with the world I have and I thought these elements would help capture what I'm going for the best

Have design goals in mind. What do you want your system to do/feel like?
Once you have those, you can check every decision you make against them to see if it works with your goals or not.

Just start, I would say. Glue those mechanics that you like together, see what works and what doesn't and work from there. The worst thing that can happen is that you decide it doesn't work at all and start again - and even then, at least you'll have gathered some experience

emsquared offered some good advice there.

What I do when I make a game (note that I might not be a great example as I tend to get bored of designing one game then move on to the next and rarely end up finishing anything...) is ask myself what am I trying to accomplish with this game? What sort of game mechanics would work well for what I'm trying to accomplish? How could I reward players for playing the game the way I intended to be played? And then I work from that.