I eat too much, or exercise too little

I've been diligently tracking my food for the past couple of weeks and for me to lose 1.5 lbs a week (my current goal), I need to eat only 1370 calories a day. I've found that that is woefully too little.

On my most successful days, I average around 1600-1700, which is nowhere near the goal. By the way, I define success as not being hungry and feeling fulfilled both physically and mentally with regards to food. For example, I don't ever want to feel deprived because I know that is not a sustainable way of life, which is my ultimate goal for this weight-loss journey.

If I want something sweet, I would like to be able to have it, even if it's just one chocolate square and not the whole chocolate bar. I've gotten much better about meting out my sweets in that I won't have the chocolate and the ice cream and the eggnog all in one day. However, I will have a small bit of one of those things because I crave sugar and don't feel good without it.

I know that's a problem that needs to be dealt with some day (once I get the exercise and portion sizes under control), but at the same time I feel like I shouldn't cut out an entire food group from my diet. I can't live without any sugar for the rest of my life and therefore I shouldn't be using that technique now because I want this lifestyle change to be sustainable.

So if having the little things that get me through the day means I will lose a little slower, so be it. I'm currently losing about a pound a week, and that's with very minimal exercise (and a significant amount of beer...also something I need to cut down on). If I can regularly get out the door for some biking or running then I think I can ramp up the rate of loss.

---
Edited to add:
My sugar intake is also in the form of apples, mangoes, oranges, etc. So I'm not just eating junk food. I simply find that I need sugary things as snacks (~80-100 calories per snack) and decently sized meals (~400 calories per meal) not to feel deprived at the end of the day. For me, deprivation is not sustainable even for the short-term, so I'm trying the slower method of weight-loss and so far it's working (slowly).

Also, the 1370 number is my net calorie goal for the day. So if I exercise, I "earn" those extra calories, which means that the more I exercise the more I can eat. On days I run (I'm doing Couch to 10k currently...again) I get about 100-200 calories more and on bike ride days I earn upwards of 700 calories.

Regardless of exercise, I try to keep my calorie count as low as I can sustain, which I've found is ~1600 calories for the day.